Nontraditional Computer Support for Design and User Evaluation - Minitrack Introduction

This mini-track is devoted to the topic of Nontraditional Computer Support for Design and User Evaluation. An objective in coordinating this mini-track was to encourage a wide range of papers that touched on the need for additional and nontraditional measures of performance for evaluating the anticipated progress as people use interactive computer systems. The potential benefit of such measurement tools and techniques has been recognized in numerous domains including education, information visualization, digital libraries, and computer supported cooperative work (CSCW). The summaries from several conferences and workshops on human-centered systems contain many references to this subject. However, many questions still remain on how and what to measure for interactive information retrieval systems. Currently, if systems are tested, designers must rely on real users to test their systems. While this constitutes an ultimate test it also raises many issues including comparison of skills of users, standardization of test situations, amount of training given, help given, and time allotted for the evaluation. Issues such as these suggest that techniques such as appropriate user modeling might assist in this type of evaluation. While the majority of researchers agree on the benefit of user interaction metrics, the form of these metrics cause considerable debate. Current performance metrics are mathematically based. However, it is likely that interdisciplinary studies will be needed to define meaningful metrics for human-computer interactions. It remains to be seen whether these metrics can be formulated mathematically. Norman [1] says: “The things we collect statistics about are primarily those things that are easiest to identify and count or measure which may have little or no connection with those factors of greatest importance. We must be careful to measure those things that are of value to the humans interacting with our systems.” Interactive systems must deliver what users want, however, this value is determined by the users. We must determine dimensions